Making Good Prediction: A Theoretical Framework

We propose approaching prediction from a framework grounded in maximizing correct prediction rates. While this is intuitively obvious, not nearly enough attention has been paid to the creation of a clear theoretical framework for prediction; motivated by the needs of current genome-wise association studies (GWAS) we provide a discussion on such a framework. We lay out an objective function for correct prediction rates for which we need to maximize, then consider and ultimately reject an estimator based on the sample analog of the solution to the maximization problem due to (1) the estimator’s inability to distinguish between noisy and predictive variables which directly leads to (2) an inability to estimate the sample analog. In response, we offer an alternative solution to the maximization problem. We demonstrate that the Partition Retention method’s I-score is a measure that asymptotically approaches this alternative solution. Based on the I-score for a given variable set we derive an asymptotic lower bound for the correct prediction rate. The I-score can differentiate between noisy and predictive variables as well. We offer simulations and applications of the I-score on real data to demonstrate its predictive performance on sample-constrained data. These show that the I-score can capture highly predictive variable sets and serves as a lower bound for the out sample correct prediction rate. Future research in the avenue of alternative solutions as sample-based measures of predictivity is much desired. ∗Department of Political Science, University of California, San Diego †Department of Statistics, Harvard University ‡Department of Statistics, Columbia University §Department of Statistics, Columbia University